going from 100 to 320 will leave lots of scratches. I refinish furniture for a living and have to sand items daily. If I sand a piece with 100, I have to then use 150, 180, 240 and finally 320 or I'll have huge sanding scratches like your pictures show.
If you sand properly, you can polish the wheels out by hand.